---
title: Crawl4AI
---

**Crawl4aiTools** enable an Agent to perform web crawling and scraping tasks using the Crawl4ai library.

## Prerequisites

The following example requires the `crawl4ai` library.

```shell
pip install -U crawl4ai
```

## Example

The following agent will scrape the content from the https://github.com/agno-agi/agno webpage:

```python cookbook/tools/crawl4ai_tools.py
from agno.agent import Agent
from agno.tools.crawl4ai import Crawl4aiTools

agent = Agent(tools=[Crawl4aiTools(max_length=None)])
agent.print_response("Tell me about https://github.com/agno-agi/agno.")
```

## Toolkit Params

| Parameter    | Type  | Default | Description                                                               |
| ------------ | ----- | ------- | ------------------------------------------------------------------------- |
| `max_length` | `int` | `1000`  | Specifies the maximum length of the text from the webpage to be returned. |
| `timeout` | `int` | `60` | Timeout in seconds for web crawling operations. |
| `use_pruning` | `bool` | `False` | Enable content pruning to remove less relevant content. |
| `pruning_threshold` | `float` | `0.48` | Threshold for content pruning relevance scoring. |
| `bm25_threshold` | `float` | `1.0` | BM25 scoring threshold for content relevance. |
| `headless` | `bool` | `True` | Run browser in headless mode. |
| `wait_until` | `str` | `"domcontentloaded"` | Browser wait condition before crawling (e.g., "domcontentloaded", "load", "networkidle"). |
| `enable_crawl` | `bool` | `True` | Enable the web crawling functionality. |
| `all` | `bool` | `False` | Enable all available functions. When True, all enable flags are ignored. |

## Toolkit Functions

| Function      | Description                                                                                                                                                                                                    |
| ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `web_crawler` | Crawls a website using crawl4ai's WebCrawler. Parameters include 'url' for the URL to crawl and an optional 'max_length' to limit the length of extracted content. The default value for 'max_length' is 1000. |

## Developer Resources

- View [Tools](https://github.com/agno-agi/agno/blob/main/libs/agno/agno/tools/crawl4ai.py)
- View [Cookbook](https://github.com/agno-agi/agno/tree/main/cookbook/tools/crawl4ai_tools.py)
